RATNER'S THEOREM AND INVARIANT THEORY 



Akihiko Yukie 



Abstract. In this note, we consider applications of Ratner's theorem to construc- 
tions of families of polynomials with dense values on the set of primitive integer 
points from the viewpoint of invariant theory. 



Let Q be a non-degenerate indefinite quadratic form in n > 3 variables. We say 
Q is irrational if the ratio of at least one pair of coefficients is irrational. 

The following theorem was called the Oppenheim conjecture and was proved in 
final form by Margulis [9]. 

Theorem 1 (Margulis) // Q(x) is an irrational non- degenerate indefinite qua- 
dratic form in n > 3 variables, the set {Q(x) | x € Z n } is dense in M. 

Many partial results were known prior to the work of Margulis (for example, 
Oppenheim [10], [11], [12], Davenport (with others) [6], [4] [5], [1], [19]). Note that 
because of a result of Lewis [7], these results imply the Oppenheim conjecture for 
such Q with n > 21. For forms of higher degree, there is a result by Schmidt [22]. 

Raghunathan reduced the Oppenheim conjecture to the following problem. 

Problem 2 Show that SO((3)rSL(3)^ is dense in SL(3)r. 

What Margulis did was to solve the above problem, and in fact he proved that 
the values of Q on the set of primitive integer points are dense in R. 

This problem can be viewed as a problem on orbit closures in Lie groups. We 
first recall Ratner's theorem. 

Theorem 3 (Ratner) Let G be a connected Lie group and U a connected subgroup 
of G generated by unipotent elements of G. Then given any lattice Y C G and 
x £ G, there exists a connected closed subgroup U C F C G such that UxT = FxT. 
Moreover, F/(F fl (xTx~ r )) has a finite invariant measure. 

The first statement was called Raghunathan's topological conjecture, and the 
second statement was proved by Ratner in conjunction with Raghunathan's topo- 
logical conjecture. Raghunathan's topological conjecture was published by Dani [3] 
for one dimensional unipotent groups and was generalized to groups generated by 
unipotent elements by Margulis [8]. The proof for the general case was given by 
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Ratner in a series of papers [14], [15], [16], [17]. For these, there is an excellent 
survey article by Ratner [18]. 

Note that F in the above theorem is unique. To see this, one should note that 
the image of F in G/T by the map g — > gxT is F/ (F fl (xTx -1 )) and F fl (xTx -1 ) 
is a discrete subgroup of F. So the tangent space of F at the unit element is 
determined by the image of F in G/T. Also if Ti,]^ are commensurable lattices 
and UxTi = FixT for i = 1, 2, Fi = i 7 ^. To see this, one can assume that Ti C T2 
and the index of Ti in T2 is finite. Then if g±, ■ ■ ■ ,gi are representatives of Ti \ T2, 

UxY 2 = U l i=1 UxT igi = U l i=1 F lX r igi = F lX T 2 . 

Because of the uniqueness of F, F 2 = F\. 

It is easy to deduce Problem 2 (and hence Theorem 1) from Ratner's theorem. 
We will not give the proof here, but we will use a similar argument later in Theorems 
7,14 to construct a family of quartic forms in four variables whose values on the set 
of primitive integer points are dense in EL 

For any algebraic group G defined over a subfield of R, let G^ + be the identity 
component of Gr in the classical topology. We denote the identity component of G 
in the Zariski topology by G°. Therefore, G^, is the set of M-rational points in G° , 
and this group may not be connected in the classical topology. If G is an algebraic 
group over C, it is connected in the Zariski topology if and only if Gc is connected 
in the classical topology. So the identity component of Gc in the classical topology 
is an algebraic group over C. We use the notation G^ for this identity component 
and this can be regarded as the set of C-valued points in the identity component 
G° in the Zariski topology. If F is an algebraic group over C and A; is a subfield of 
C, we say F is defined over k if there exists an algebraic group G over k such that 
F = Gc- Suppose G is an algebraic group defined over a subfield k of C. Then it 
is known that the identity component G c is defined over k (see [2, p. 47]). 

If a connected Lie group F is in the form F = G^ + for an algebraic group G 
over Q, we say that F is defined over Q. Moreover, we use the notation Fc for Gc 
even though this is a slight abuse of notation. By the above remark, G c is defined 
over Q. So we may choose such G so that Gc is connected. This G is automatically 
connected in the Zariski topology over any extension of Q. 

Now let Hi C H 2 be algebraic groups over Q (and so Hi is a closed sub- 
group). We assume that H 2 is semi-simple, and Hi£,H 2 £ are connected. Consider 
H° R+ , H% R+ . For any closed connected Lie subgroup H C -#2R+> we consider the 
following condition. 

Condition 4 H is generated by unipotent elements. Moreover, the projection of 
H to any simple component of i?2R+ is non-trivial. 

Note that if a connected Lie group H is simple and non-compact, H is generated 
by unipotent elements. We assume that H° R+ satisfies Condition 4. Suppose we 
can carry out the following problem. 

Problem 5 (a) Classify all the closed connected complex Lie subgroups F such that 
Hie C F C H 2 £ and show that there are only finitely many such F's. Moreover, 
show that all such F's are defined over Q. 
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(b) For each F ^ H 2 £, construct an ^-variety Xp denned over Q with the property 
that F has a unique fixed point wp G Xfq in Xp£. 

Of course, we may not always be able to carry out Problem 5. Note that the 
existence of Xp in (2) in many cases follows from a theorem of Borel and Harish 
Chandra. More precisely, suppose F in Problem 5 satisfies the following condition 

{h e H 2C | hfh- 1 G N H2C (F) for all f e F} = N H2C (F) 

(N# 2C (F) is the normalizer of F in H 2 c)- Then there exists a faithful representation 
H 2 ->■ GL(E) of H 2 defined over Q and a point v G F(E)q such that N H2C (F) is 
the stabilizer of v in H 2 c (see 5.1 Theorem [2, p. 89]). Let X be the image of 
H 2 in P(£') by the map H 2 3 h -> G P(£). Then X is a locally closed subset 
of P(E') defined over Q (since X is a homogeneous space). Then y = hv G Xc 
(/i G H 2 c) is a fixed point of F if and only if hfh -1 G N# 2C (.F) f° r an f € F, an d 
so /i G N_e- 2C (F). So we may choose this X as Xp. We will consider Problem 5 for 
certain situations (see Problem 9), and so far the above condition is satisfied for all 
the cases we considered. 

In the following, we assume that we carried out Problem 5. 

Definition 6 An element g G H 2 £ is sufficiently irrational if gwp ^ Xpq for all 
F 7^ H 2 £. 

Note that if F±,F 2 are such groups and there exists an i^-equivariant map 
Xp 1 — >■ Xp 2 defined over Q, we don't have to consider Xp 1 , because Xp x q maps to 
Xp 2 q- Let T C F[% R+ be an arithmetic lattice. Now the following theorem is an 
easy consequence of Ratner's theorem. 

Theorem 7 Suppose g G H 2 £ is sufficiently irrational, gHicg -1 is defined overM., 
and the identity component U of (gHicg -1 ) n i?2K satisfies Condition 4. Then UT 
is dense in #2R+- 

Proof. By Ratner's theorem, there exists a closed connected subgroup U C F\ C 
-^2R+ sucn tnat = -^1^- Moreover, Fi is defined over Q by Proposition (3.2) 
[23, pp. 321-322]. Note that Shah considered a particular lattice in [23], but by 
our commensurability discussion of lattices after Ratner's theorem, we can consider 
any arithmetic lattice. 

Suppose Fi ^ i?2K+- Then dim.Fi < dimH^^- Since both groups are defined 
over Q, Fic ^ H 2 £ by considering the Lie algebras. Since gHicg~ x is defined over 
M, gHicg~ x C Fic- So Hie C g~ 1 Ficg ^ H 2 £. Since we are assuming we carried 
out Problem 5, there exists an algebraic group F 2 over Q such that g~ x Ficg = F 2 £. 
Since Wp 2 is the unique fixed point by g~ 1 Ficg, Pf± = gwp 2 is the unique fixed 
point by F 1C . Let a G Aut(C/Q). Then F 1C = F? c clearly fixes p Fi . By the 
uniqueness of the fixed point, p Fi = pf l . Since this is true for all a G Aut(C/Q), 
Pf 1 = gwp 2 G Xf 2 q, which is a contradiction since we assumed that g is sufficiently 
irrational. □ 

We want to use the above observation in the setting of prehomogeneous vector 
spaces. We now recall the definition of prehomogeneous vector spaces. Let G be a 
reductive group, V a representation of G, and x a non-trivial character, all defined 
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over a field k of characteristic zero. We assume that G^ is connected in the Zariski 
topology. 

Definition 8 (G, V, x) is called a prehomogeneous vector space if 

(1) there exists a Zariski open orbit, 

(2) there exists a non-zero polynomial A(x) 6 k[V] such that A(gx) = x(g)A.(x) 
for all g <G G, x e V. 

Let F ss = {i 6 7 | A(i) / 0} and we call it the set of semi-stable points. A 
polynomial A(x) in (2) is called a relative invariant polynomial. If the represen- 
tation is irreducible, the choice of x is essentially unique and we may write (G, V) 
also. 

For the rest of this paper, we only consider irreducible prehomogeneous vector 
spaces. Let (G, V) be a prehomogeneous vector space. If there exists a relative 
invariant polynomial A(x) whose Hessian is not identically zero, (G,V) is called 
regular. It is known (see [21]) that the regularity is equivalent to the existence of a 
point w in the open orbit in V fc ss whose stabilizer G w is reductive (since ch k = 0). 
An important consequence of the regularity is that V^ ss is a single G^-orbit. Up to 
a certain notion of equivalence, irreducible regular prehomogeneous vector spaces 
over any algebraically closed field k of characteristic zero were classified by M. Sato 
and Kimura in [21]. Each equivalence class contains a prehomogeneous vector space 
(G,V) such that dimF is the smallest, and such (G,V) is called reduced. Over 
local and global fields, the classification was carried out by H. Saito [20]. We list the 
split forms of irreducible reduced regular prehomogeneous vector spaces as follows. 

(1) G = GL(n) x G' , V = M(n, n) where G' C GL(n) is a reductive subgroup such 
that k n is an irreducible representation of G' . 

(2) G = GL(1) x GL(n), V = Sym 2 k n . 

(3) G = GL(2ra), V = A 2 k n . 

(4) G = GL(1) x GL(2), V = Sym 2 k 3 . 

(5) , (6), (7) G = GL(1) x GL(n), V = A 3 k n where n = 6, 7, 8 (as in E 6 , E 7 , E 8 ). 

(8) G = GL(3) x GL(2), V = Sym 3 <g> k 2 . 

(9) G = GL(6) x GL(2), V = A 2 k 6 ® k 2 . 

(10) , (11) G = GL(n) x GL(5), V = k n ® A 2 A; 5 where n = 3, 4. 

(12) G = GL(3) x GL(3) x GL(2), V = M(3, 3) ® ^ 2 . 

(13) G = GSp(2n) x GL(2m), V = & 2n (g> ^- 2m where n > 2m > 2. 

(14) G = GSp(6) and V = k 1A is the irreducible representation which corresponds 
to the third fundamental weight. 

(15) G = GO(n) x GL(m), V = A; n ® k m . 

(16) , (17), (18) G = GSpin(7) x GL(n), V = k 8 ® k n where k 8 is the spin repre- 
sentation and n = 1, 2, 3. 

(19) , (22) G = GSpin(n) where n = 9, 11 and V is the spin representation. 

(20) , (21) G = GSpin(lO) x GL(ra), V = A; 16 (g> fc n where fc 16 is the half spin 
representation and n = 2,3. 

(23), (24) G = GSpin(n) where n = 12, 14 and V is the half spin representation. 
(25), (26) G = G2 x GL(n), V = k 7 ® k n where G2 is the automorphism group of 
the split octonion, k 7 is the imaginary part of the split octonion, and n = 1,2. 
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(27), (28) G = E 6 x GL(n), V = J ® k n where E 6 is a semi-simple group of type 
E6 and J is the exceptional Jordan algebra. 

(29) G is a group whose semi-simple part is a group of type E7 and V = k®k®J®J. 

In the above list, GSp(2n), GO(n), GSpin(n) are the groups defined similarly as 
Sp(2n), O(n), Spin(n) except for allowing scalar multiplications. 

Now let (G,V) be an irreducible prehomogeneous vector space in (l)-(29). Let 
w G V£ s , and H = [G, G]. Then H is a semi-simple group. Let H w be the stabilizer 
of w in H. We consider the following two problems. 

Problem 9 (1) Carry out Problem 5 for Hi = #° , H 2 = H. 

(2) Carry out Problem 5 for the image Hi of H in SL(F) and H 2 = SL(V r ). 

For Problem 9(2), we do not necessarily have to consider prehomogeneous vector 
spaces. Let V be a rational representation of a semi-simple algebraic group H over 
Q such that He is connected. Then we can consider Problem 9(2) for Hi, H2 defined 
in the same manner. 

Consider problem 9(1) for the case (1) in the classification. Let H' = [G',G']. 
Then H = SL(n) x H' in this case, and w = I n (the identity matrix) belongs to 
V| s . It is easy to see that H w = {(h, t h~ 1 ) \ h G H'}. So Problem 9(1) for this case 
is almost equivalent to Problem 9(2) for irreducible representations of semi-simple 
groups. Since this case contains a rather large class of representations, we do not 
consider the case (1) in the classification when we consider Problem 9(1), and treat 
it separately as Problem 9(2). 

Consider Problem 9(1). If x G Vj| s , there exists g G Gc such that gw = x. We 
can choose t in the center of Gc and h G He so that g = th. 

Definition 10 A point x G Vj| s is sufficiently irrational if there exists an element 
g = th e Gc as above such that h is sufficiently irrational. 

Problem 9(1) is not applicable to cases (2) n = 2, (4), (8), (9), (11), (12), (15) 
m = 2, (17), (26), (28) because stabilizers of points in V ss do not satisfy Condition 
4. In [26], [25], [27], we considered cases (3), (5), (6), (7), (10). In [28], we consider 
all the other applicable cases except for the cases (1), (29). For the case (29), we 
seem to need a new interpretation of the "Freudenthal quartic". For Problem 9(1), 
we also construct G-equivariant maps 4>p '■ V ss — > Xp for all F's in Problem 5 so 
that 4>f(w) = wp- Moreover, the center of G acts trivially on Xp for all F. So 
an alternative way of stating the above definition is that x e Vj| s is sufficiently 
irrational if 4>f(x) ^ Xpq for all F, because 4>f(x) = t(f>F(w) = twp. 

If x e V^ s , there exists g 6 Gc such that x = gw. We write g = th as above. 
Then H x c is defined over M. Since H° c = hH^h -1 , this group is also defined over 
R. Note that we cannot assume g e Gk in Definition 10 since V^ s is not necessarily 
a single GR-orbit. 

We state an application of Problem 9(1). Consider the case (5) in the classifica- 
tion. Let ei, • • • , eQ be the standard basis of k 6 , and w = ei A e2 A e% + e.4 A es A e^. 
Let Ei, be subspaces of k 6 spanned by {ei, e<i, ez\, {e^, e$, ee} respectively. It 
is known that w € Vj| s . Let Gr(3,6) be the Grassmann of 3-spaces in k 6 , and 
X = (Z/2Z) \ (Gr(3,6) x Gr(3,6)) where Z/2Z acts by exchanging two factors. 
Let [Ei], [E 2 ] E Gr(3,6), ([Ei], [E 2 ]) € X be the points determined by E ± ,E 2 . Let 
x = Y2 x ijk e i A ej A = gw G X^| s where G GL(6)c- Then it turns out that 
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x is sufficiently irrational if g[Ei], g[E 2 ] £ Gr(3,6)Q and g([Ei], [E 2 ]) ^ Xq. Both 
Gr(3, 6) and X are defined over Q and one can determine whether or not a point 
in these spaces is rational by considering Pliicker coordinates. In this case, H° R+ 
satisfies Condition 4 for all x € V^ s [25]. The following result is proved in [25]. 

Theorem 11 Suppose x e Vj| s is sufficiently irrational. Then for any y = (yijk) £ 
A 3 R 5 and e > 0, there exists a Z-basis {ui, ■ ■ ■ ,uq} of (Z 6 )* (the dual) such that 

\y ijk - x(ui,Uj,u k )\ < e 

for all i < j < k < 5. 

We proved similar statements for cases (3) , (6) in [25] (the irrationality condition 
is slightly easier). 

We list Xp's for Problem 9(1) except for the cases (1), (29). 

(2) P(V) 

(3) F(V) 

(5) Gr(3, 6), Gr(3, 6), (Z/2Z) \ (Gr(3, 6) x Gr(3, 6)) 

(6) P(Sym 2 (Q 7 )*) 

(7) P(Sym 2 (Q 8 )*) 



10 
13 
13 
14 
15 
15 
16 
18 
19 
20 
21 
22 
23 
24 
25 
27 
29 



P(Sym 2 (Q 3 ) * ) , P(Sym 2 (Q 5 ) * ) 
(2m ^ n) Gr(2m, 2ra), P(A 2 (Q 2m )*) 

(2m = n) (Z/2Z) \ (Gr(2m, 4m) x Gr(2m, 4m)), P(A 2 (Q 2m )*) 
the same as (5) 

(2m ^ n) Gr(m,n),P(A 2 (Q m )*) 

(2m = n) (Z/2Z) \ (Gr(m, 2m) x Gr(m, 2m)), P(A 2 (Q m )*) 
F(V) 

Gr(3,7),P(Sym 2 (Q 3 )*) 
P 8 

Gr(3, 10) 

(Z/2Z) \ (Gr(5, 10) x Gr(5, 10)), Gr(4, 10), P 9 , P(Sym 2 (Q 3 )*) 

Gr(6, 12), Gr(6, 12), (Z/2Z) \ (Gr(6, 12) x Gr(6, 12)) 

the same as (22) 

(Z/2Z) \ (Gr(7, 14) x Gr(7, 14)) 

F(V) 

F(V) 

? 



We probably need some explanation. For (5), we can associate two 3-dimensional 
subspaces of a 6-dimensional space (which is the standard representation of GL(6)) 
to any point in V ss . For this, the reader should see [25]. For (10), we can associate 
quadratic forms on two vector spaces V\ , V 2 of dimensions three and five (the preho- 
mogeneous vector space (5) is considered as A 2 V2®Vi) to any point in V ss . For (18), 
we can associate a 3-dimensional subspace of a 7-dimensional space (which is the 
vector representation of GSpin(7)) to any point in V ss . For (19), P 8 is the projec- 
tive space associated with the vector representation of GSpin(9). For (21), we can 
associate a 4-dimensional subspace, a 5-dimensional subspace of a 10-dimensional 
space (which is the vector representation of GSpin(lO)), and a quadratic form in 
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three variables to any point in V ss . By considering combinations of these subspaces, 
we get varieties as above. For (20), (22), (24), the numbers 10, 12, 14 are the di- 
mensions of the vector representations. For (13) (2m = n), (15) (2m = n), (20), 
(24), we do not have to consider Gr(2m, 4m), Gr(m, 2m), Gr(5, 10), Gr(7, 14) as we 
do for (5), (14), (22). 

Consider Problem 9(2). Let (G,V) be a prehomogeneous vector space in (1)- 
(29), and A(x) the relative invariant polynomial of the minimum degree. Suppose 
we carry out Problem 9(2). Then we conjecture the following application. 

Conjecture 12 If g 6 GL(V)k is sufficiently irrational, the values of A(g~ 1 x) on 
the set of primitive integer points x are dense in R. 

Note that if f(x) is a polynomial on V, the natural action of g on / is given by 
f(g~ 1 x). Let d be the degree of A(x). Then since (G, V) is a prehomogeneous vector 
space, A(x) is the unique He invariant of degree d up to a constant. Therefore, 
we may choose X F = P(Sym d F*) and w F = [A] G P(Sym d V*) (the class of A) for 
F = H 1C for Problem 9(2). 

We will carry out Problem 9(2) and Conjecture 12 for the case (4) in the clas- 
sification here. In this case, H = SL(2) and H2 = SL(4). If g is in the kernel of 
the homomorphism H — >■ H2, g is a scalar matrix tl 2 and t 3 = 1. Since this is an 
element of SL(2), t 2 = 1 and this implies t = 1. So we may regard H as a subgroup 
of H 2 . Therefore, H\ = H = SL(2) in this case. 

We regard V as the space of cubic forms in two variables v = (vi,v 2 ). For 
x = x vf + x\v\v 2 + x 2 v\v\ + x 3 v 2 , y = yovf + yiv 2 v 2 + y 2 viv 2 + y 3 v 2 , we define 

B(x, y) = x y 3 - -xit/ 2 + 3^2^/1 - x 3 y . 

Then it is well known that the alternating bilinear form B is invariant under the 
action of Hi. For example, this B is used in [24, p. 154]. We denote the symplectic 
group with respect to B by Sp(4). 

Lemma 13 // Hie C F C H 2 c is a closed connected complex Lie subgroup, F = 
SL(2) c ,Sp(4)c, or SL(4) C . 

Proof. Let f)i, f, {) 2 be the Lie algebras of -Hie, F, H 2 £. These are vector spaces over 
C. Let A be the fundamental dominant weight of the Lie algebra f)i. We use the 
notation dA for the irreducible representation of f)i with highest weight dA. 

By considering weights, f) 2 = t)i © 4A © 6A and f)i = 2A. Let U x = 4A, U 2 = 6A. 
Let f)3 = sp(4)c be the Lie algebra of Sp(4)c- Counting the dimension, ty 3 = {]\®U 2 . 
Since {) 3 is a Lie subalgebra of ty 2 , {) 2 is a representation of {) 3 . Since f)3 is semi- 
simple, there exists a representation U 3 of f)3 such that {) 2 = t) 3 © L^. Now ^3 can 
be regarded as a representation of f)i. Since ty 3 is isomorphic to 2A © 6A, U 3 must 
be the factor 4A. So U\ = 4A is a representation of f)3. This implies that [Ui, U\] 
is a representation of {) 3 also. Since {) 2 is a simple group, [Ui,Ui] fl f)3 7^ 0. Since 
f)3 is the adjoint representation of a simple group, it is irreducible, and therefore, 
[Ui, Ui] D f)3. So we can conclude that if f D Ui, f = {) 2 . This proves the lemma. □ 

The groups SL(2), SL(4), Sp(4) are all defined over Q. It is known [24, p. 150] 
that the relative invariant polynomial of the lowest degree is 

A(x) = x\x\ + \%xqX\x 2 x 3 — 4:Xqx 2 — 4x^3 — Tlx\x\ 
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for x = xovf + Xivfv2 + X2V1V2 + x^V2- Therefore, for F = SL(2)c, we may choose 
X F = P(Sym 4 V*) and w f = [A] (the class of A). It is well known that the 
alternating bilinear form fixed by a simplectic group is a constant multiple of the 
corresponding alternating bilinear form. Therefore, for F = Sp(4)c, we may choose 
X F = P(A 2 F*) and w F = [B] (the class of B). This is the answer to Problem 9(2) 
in this case. 

Theorem 14 Suppose g G GL(4) R , g[A] £ P(SymV*) Q , and g[B] <£ F(A 2 V*) q . 
Then the values of A(g~ 1 x) on the set of primitive integer points x are dense in R. 

Proof. If detg < 0, we choose g 1 G GL(4) Z such that det g' = -1. Then g[A] £ 
P(SymV*) , g[B] £ P(A 2 F*) Q jf and only if ^[A] £ P(SymV*) Q , g'g[B] £ 
P(A 2 F*)q and the values of A(g~ 1 x) on the set of primitive integer points x are 
dense in R if and only if the values of A(g~ 1 g'~ 1 x) on the set of primitive integer 
points x are dense in R. Therefore, we may assume that detg > 0. Then we 
can write g = tl 4 g" where t G l x and g" G SL(4) K . Since g[A] = g"[A] and 
g[B] = g"[B], we may assume g e SL(4)r. Note that the center of SL(4) acts on 
P(Sym 4 F*),P(A 2 F*) trivially. 

Let T = SL(4)z. Clearly, Hm, Hzr are connected in the classical topology. So, 
by Theorem 7, (gHu^g -1 ^ is dense in H 2 r. This implies that T(gHiK_g~ 1 ) is dense 
in H 2 r also. Let rel x . Let diag(£i, t2, ^3, t^) be the diagonal matrix with diagonal 
entries ti, ^2,^3, ^4- Then h = diag(— |, — |, 1, 1) e SL(4)r and 

A(/i" 1 (l,0,l,0)) = r. 

There exist hi G gHmg -1 and /12 G T such that /i2^i is close to hg -1 . Then 

(fc 2 fci0A)(l, 0, 1, 0) = (h 29 A)(l, 0, 1, 0) = A(p-^2 '(1, 0, 1, 0)) 

is close to /iA(l, 0,1,0) = A(/i" 1 (l, 0, 1, 0)) = r. Since h^{l, 0, 1, 0) is a primitive 
integer point, this proves the theorem. □ 

Theorem 14 gives us a 13-dimensional family of quartic forms in four variables. 

Question 2 can be considered as Problem 9(1) for the prehomogeneous vector 
space (2) n = 3 or Problem 9(2) for the prehomogeneous vector space (2) n = 2. 
All the analytical difficulties are contained in Ratner's theorem and Problem 5 for 
this case is almost trivial (which of course does not mean that Question 2 is trivial) . 

As far as Problem 9(2) is concerned, besides the prehomogeneous vector space 
(4), we carried out this program for the representation (SL(2), Sym 4 A; 2 ) [27], and 
the adjoint representation of SL(3) [26]. 

Let V be the representation space for both cases. For these cases, dim V = 5 or 
8. It is well known that there are a quadratic invariant Q(x) and a cubic invariant 
F(x) for both cases. Then we have the following theorem [26], [27]. 

Theorem 15 For the above two cases, if g G GL(y)R and Q(g~ 1 x) is irrational, 
the values of F(g~ 1 x) at primitive integer points are dense in M. 

These are a 22-dimensional and a 56-dimensional family of cubic forms in 5 and 
8 variables respectively. For cubic forms, Pitman [13] proved a weaker form of 



RATNER'S THEOREM AND INVARIANT THEORY 



9 



the Oppenheim conjecture (in the sense that if f(x) is the form and e > 0, there 
exists an integer point x 7^ such that < e). Still this weaker form requires 

(1314) 256 variables, and it is of interest to construct examples of forms with dense 
values at primitive integer points. 

Consider Conjecture 12 for the prehomogeneous vector spaces (l)-(29). If the 
degree of A is two, the conjecture is not so interesting because it is contained in the 
Oppenheim conjecture. This applies to cases (16), (19), (25). If a prehomogeneous 
vector space (G±, V\) is contained in another prehomogeneous vector space (G2, V2) 
and the restriction of a relative invariant polynomial on V 2 is a relative invariant 
polynomial on Vi, Conjecture 12 for (G 2 , V2) is not so interesting. This applies to 
cases (1), (5), (12), (17), (22), (23), (26), (27), (28), (29), because the cases (1), 
(27) reduce to the case (2), the cases (5), (22), (23), (29) reduce to the case (14), 
the cases (12), (28) reduce to the case (8), and the cases (17), (26) reduce to the 
case (15) m = 2. It is not entirely clear, but probably the case (3) reduces to the 
case (2), the case (9) reduces to the case (8), and the case (13) reduces to the case 
(15). This leaves us the following cases. 

case (2) (4) (6) (7) (8) (10) (11) (14) (15) (20) (21) (24) 
degA n 4 7 16 12 15 40 4 2m 4 4 8 

Of course it is more interesting if deg A is relatively large compared to the number 
of variables. In particular, by considering the cases (8), (11), it may be possible 
to produce families of polynomials of degree 12 and 40 in 12 and 40 variables 
respectively with dense values on the set of primitive integer points. 

Finally it is curious to see if one can construct families of polynomials with non- 
trivial moduli (in other words, they consist of infinitely many orbits) with dense 
values on the set of primitive integer points. For example, it is possible to do so 
using the representation (SL(2), Sym 4 Q 2 ), but we have yet to know if this is possible 
for the spaces of binary forms of higher degrees for example. 

Acknowledgement The author would like to thank D. Witte for reading the 
manuscript carefully. 
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